Huddled Masses

Posts Tagged ‘PoshCode’

So, I’m building the next PoshCode around modules, with CPAN as my model …

Scripts are still there, but they’re still basically done the way they are now: you just upload a single file.

Modules, however, will be treated specially. You’ll have to have an account to upload, but you’ll be able to just upload a .zip file of your module folder using your login and not fill in any other forms. Our system will take care of parsing the metadata out of the manifest in your module folder.

The problem is that I need a little data that isn’t a part of the standard module manifest format… and I can’t add it because PowerShell won’t load a module with extra fields in the manifest metadata hashtable: https://connect.microsoft.com/PowerShell/feedback/ViewFeedback.aspx?FeedbackID=421837 — My request was closed “by design” early in the beta cycle and I wasn’t able to convince them to rethink that.

At a minimum, PoshCode requires a LICENSE field, and a CATEGORY field (think of the categories on CPAN or TechNet Script Center).

So I’m having an informal poll. (you’ll have to “vote” by commenting or tweeting to me, because I want more than just “I choose A”).

Which option do you prefer, or can you think of a better way:

Require the License and Category to be part of the PRIVATEDATA field in the standard manifest

This might require authors to rewrite parts of their modules, because we’d be forcing PrivateData to be a Hashtable, and to contain keys that they don’t need in the module.

Some existing modules use an array in PrivateData, or even a simple string, rather than a Hashtable. However, it’s not a huge difference, and might be the cleanest method: it wouldn’t require you to manage a second file of data, and the additional data will be easily available to users and scripts via the standard Get-Module output.

Add a “doc comment” system for Module manifest files.

The upside of this is that PoshCode wants to create module-level documentation anyway. If we use documentation comments like those used on functions we would be able to just create our own standard for them, and add any extra fields we need. It would be guaranteed not to conflict with anything you’re already doing, and in addition to missing metadata, you could have some module-level documentation beyond just the Description field of the metadata.

The problem with this is that there’s no built-in way to parse that, and doing so isn’t trivial, so you would need to just read it on our website, or read the source of the file, or have our PoshCode cmdlets in order to make any sense of it once you had a module on your system. It doesn’t integrate with PowerShell in any meaningful way.

Embed the extra data right in the manifest hashtable, using a special comment to hide it from Get-Module

@{
author=“Joel Bennett”
description=“My latest crazy module”
<#!PoshCode
License=“Ms-PL”
Category=“WPF”,“GUI”,“Toolkit” #>
CompanyName=“Huddled Masses”
...
}


I like this because it’s fairly trivial for us to strip out the comment and just turn that into a plain-old DATA section. It also lends itself to reminding the PowerShell team that these fields are missing, and maintains the existing simple syntax of the manifest.

The problem here is, again, that the data doesn’t appear using any of the standard PowerShell tools — but getting it out is significantly easier than getting out document comments…

Require a separate “ReadMe.psd1” (or “PoshCode.pds1” or “Metadata.psd1”) metadata file

You could start with a copy of your module manifest, and then add the extra stuff that PoshCode needs. This would be nice because we would be able to add any extra fields we wanted as mandatory members, and we could include the “documentation” stuff I mentioned earlier…

But the downside is that it would be a third file (second manifest) that authors would have to maintain and keep current.

Any thoughts? Suggestions?

I’ve put some further thoughts about CPAN and the data problem on the wiki.

Well, over the weekend I stole a few moments from thinking about PoshCode 2.0 to think about PoshCode 1.0 … and I added two things:

Feedback

I’ve replaced the misused “comment” box on the website with a full GetSatisfaction.com widget, so you can get your voice heard and get feedback (in the past it’s been impossible for me to respond to comments).

Twitter

PoshCode now posts new contributions to twitter on it’s own PoshCode account. Feel free to follow along. Of course, the information on there is severely truncated, by comparison with the PoshCode RSS feed … and only includes the link to the website, whereas the RSS feed also includes direct download links … but I figured some of you would appreciate it anyway, and I aim to please.

Of course, now in PoshCode 2 I’ll have to make sure we have users enter their twitter id’s in the profile page so we can be sure to cite you properly. Oh what a tangled web we weave…

I’ve updated the PoshCode script module to support CTP3, and added a -limit parameter to the Get-PoshCode cmdlet so you can specify how many items you want retrieved in the case where there are a lot of matches for your search terms — by default the limit is 25.

Improvements to the underlying web search API

You’ve always been able to pass a LIST parameter to the API, and get more results by specifying a higher number. But it never worked with the “path” notation (until now).

That is, you used to be able to do:

• http://poshcode.org/api1?q=start&list=10

• http://poshcode.org/api1?q=start&list=100

To make the API a little easier to use I’ve enhanced it just now:

1. You can now page the search results.
2. You can use the word “limit” instead of “list”
3. If you specify limit=0 (or list=0) I’ll give you everything I’ve got. Please use a little precaution about that, as it could be a LOT of data. I’d much rather you retrieve, say … 25, and then get the second page if you want more.

1. You can use path notation.

So, you can use any of these URLs:

• http://poshcode.org/api1/start/list/25/page/1
• http://poshcode.org/api1/start/limit/25/page/2
• http://poshcode.org/api1?q=start&list=25&page=3

• http://poshcode.org/api1?q=start&limit=25&page=4

There are a lot of search results for “start” ... feel free to play with enhancing the PoshCode module, or incorporating this into your apps, etc.

[new] I should add that you don’t have to specify the limit or page number. By default you’ll get the first 10 items, which should be enough.

I keep forgetting to mention this! I’m speaking at the next PowerShell Virtual User Group this Thursday, November 13th at 12PM EST. Join us if you’re interested in any of my PowerShell CodePlex projects (PoshConsole, PoshCode, WASP). Actually, join us even if you’re not, because there’s going to be a demo of the latest version of PowerShell Plus by the folks from Idera, and Nathan Winters will be discussing managing Exchange with PowerShell.

Also, the PowerShell Community is running a survey about the possibility of a PowerShell convention, and what you’re interested in seeing at one… it’ll just take a moment

Well, an interesting comment came up today that the new version of PoshCode.org that we’re working on should support OpenSearch and of course, me being me, I said: oh, OpenSearch is easy, gimme a minute…

And three hours later, I’m going to bed.

However, I did manage to add paging support so you can “browse” the repository, and of course … to add OpenSearch. If you visit the repository with an OpenSearch aware browser, like say … Firefox 3, or IE 7, you’ll get a little glowing notification on your search bar that there is a search engine available. You can quickly add PoshCode’s search to your browser with a couple of clicks, and in no time flat you’re searching for PowerShell script.

Incidentally, I also hooked the OpenSearch stuff up to our API output, so OpenSearch aware apps which do not simply render the HTML page can get RSS output that should work for you. We’re not doing all of OpenSearch at this point, but I think the important bits are there (certainly the bits supported in your web browser are).

Feedback is welcome as always, let us know what you think of the new stuff by emailing feedback at poshcode …

So, there’s a few new features in the latest version of the PoshCode script module, and a few new features in the repository itself, including an embed feature, which I will demonstrate by embedding one here. Notice that the scripts on the site have an embed code, and it will normally embed the full height of the script, but you can add a height parameter as I did here to force the height and cause vertical scrolling.

Resolve URL

New Cmdlet features

The latest version of both the v1 compatible PoshCode script (you have to dot-source it) and the v2 CTP2 compatible module support an -Upgrade parameter to the Get-PoshCode script, so you can call Get-PoshCode -Upgrade and it will retrieve the latest version of the script (whether you need it or not).

The New-PoshCode script has been fixed so you can use it to submit to other pastebin sites (particularly, the temporary one we use for support on IRC).

New Site features

The searches now return items with the same “probability” of being correct in date order, so newer versions of scripts show up at the top — this also affects the cmdlets, and the integration with apps such as PowerGUI. Note, however, that if an older version of the script fits your search-terms better, it may still show up at the top — I’m working on this.

The PoshCode site now supports wrapping by other community sites — if you want to wrap our site in yours, let me know, and I can give you a custom subdomain and allow you some control over the display, as I did for PowerShell Community …

I finally fixed the problem with logging that my host was having, so I have a week’s worth or so of logs which Webalyzer informs me show that we average about 600 visits a day, and a surprising number of subscribers to our RSS feed…

Upcoming features

I have three new features planned for the next couple of weeks or so: finishing the browsing feature, implementing better version-tracking, and adding tagging. The key feature missing from the brownsing is the ability to go to the “next page” of results (right now you can only see the first page of scripts for the site or for authors) — we should have that fixed soon.

The version tracking feature (which is mostly a matter of cleaning up some db queries) will result in being able to build a visual history for a given submission, including branching, etc. and should allow users to easily retrieve “the latest” version of a given script.

Tagging will improve searching and browsing by providing a way to categorize scripts and browse by tags, but I’m a little nervous about it because it seems that to be useful it will need to allow tags to be added by users who didn’t create the original script, and this is complicated since we have not so far created a login/authentication system.

I’m still debating how to properly do this, but I think that ultimately we’re going to have to use a login system with support for OpenID so that authors can identify themselves with their blog URLs, etc … and can use the same identity on the PowerShell Community site once that site gets their OpenID implementation working properly.

The alternative that I had been considering: using signed scripts as the only authentication, seems to fail in the instance of tagging — it would basically require you to make up a PSD1 file specifying the script id and the tags you want to apply and then sign it and submit it. I think you’ll agree that’s way too much work for the end users, even if it was automated — and it would also require the server to validate the signature and parse the script!

I’ve been having problems with the search functionality on the PoshCOde repository, and I just thought I’d throw this up here because I just now solved the biggest problem: ranking. Up until now, the results have not been returned in order of relevance — this is because the search works using MySQL’s FULLTEXT BOOLEAN search, which doesn’t return in relevance order, nor does it return an extra ‘score’ column.

I’ve fixed that, and weighted the search so that words in the title count more than words in the code by creating a relevance column by hand:

Incidentally, the FULLTEXT index means that words shorter than 4 characters don’t count (I’m going to try to get this changed, but it’s an option for MySQL, so it has to be changed in the config file) in the meantime you can search for words using the wildcard character, like: SQL* and it sort-of works. The PoshCode cmdlet actually was adding *‘s to the query (although I’ve just decided that’s not a good idea, because it means that queries from the cmdlet appear to have different results than queries on the website.

MySQL’s FULLTEXT BOOLEAN search has all sorts of features (and limitations): there is a stopword list, maximum and minimum word lengths, and all sorts of operators for setting word precedence or negating words, or weighting them negatively … to REQUIRE that a word be present, it must have a + in front, and in order to mark a word as more important, you have to put > in front, not just put it first… I’ve been thinking about trying to apply a few of those tricks myself (eg: put * on words under four characters, and put > on the first 30% of words and < on the last 30% to try to simulate weighting them …) but my original feeling was that the search is more powerful if you just know that it’s a fulltext boolean search and can write your queries accordingly.

If anyone has any ideas for how to improve search in MySQL … or opinions on whether I should try to apply boolean operators to queries which don’t already have them … please let me know.

Well, after some extensive hacking this long weekend, I’m happy to announce the opening of the refurbished (and renamed) PowerShell Code repository at it’s new domain: PoshCode.org with a few new features and a slightly overhauled look.

I expect that soon we’ll be replacing the back end with something which has a few enhancements over this version (at least in terms of tracking the many versions of a particular code contribution), but even without that, we hope that the current feature set will be enough to push the repository into mainstream usage as the primary place to share PowerShell code.

The new PoshCode site is still based on the old pastebin GPL source code, but I’ve hacked it up so much you’ll hardly recognize it. Sadly, I haven’t kept it as generically clean as I could have, so although I’m making the source code available, you’ll have to clean up the “layout” files if you want to run your own repository.

New Features

The most obvious new feature is that the front page now prominently features the most recent contributions. This list is filtered to only show the last in a series of contributions (so if you make a new contribution based on a previous one, the front page won’t be swamped with iterations of a single script the way the old one was).

The best new feature is that you can now browse by author (currently, you simply get a list of their 25 most recent contributions, in the future we’ll add some paging functionality as the need arises).

Finally, I’ve changed the description field (which previously allowed any html) to use only a restricted subset of Textile. I’ll put up some documentation about this later, but for now: you can make paragraphs by leaving a blank line, and can use asterisks for bold and underscores for italics … and starting each line with an asterisk or a pound sign will get you bulleted or numbered lists. Any links will be rel=“nofollow” and you can’t use raw HTML at all.

Earlier Enhancements

In addition to these new features, there are a few other enhancements that I added on to the original code: a couple of extra fields, permanent storage, search functionality, and a recent contributions RSS feed.

Incidentally, the search functionality allows you to search the author, description, title, and main content fields using MySQL’s full-text indexing search, and is accessible via the new PoshCode script functions (available as both a PowerShell v2 CTP-compatible PoshCode module and a v1 PoshCode script).