Posts Tagged ‘PoshCode’

OK, I’m putting this out there to see if I’ve missed anything that would be useful. This is the list of metadata that we’ll be storing about Modules in the PoshCode repository. It duplicates most of the information in the PowerShell Manifests, but it adds more of information that we feel is important for searching, sorting and organizing … and in the PoshCode Module manifests, wildcards are not allowed for any of the “export” values. The export information in the module is intended to represent the initial state of the module after import: listing all the cmdlets, functions, variables and aliases …

PoshCode Module Manifest Format:

GUID

The globally unique identifier for this module. Helps us tell modules apart

Name

The name of this module. Helps us find modules, and helps you guess what they’re for.

ModuleVersion

The version of this module. Together with the GUID, this is the unique key for the module. It should be incremented whenever you release it.

Description

A short description of the module. What it is, what it does, what it’s for.

ModuleType

Tells whether the module is a Script module or a Binary module (if they are both, then they are Binary).

Author

The person(s) who created this module. No organizations here, just people.

Company

A company (if there is one) that created this module or provides support for it.

Copyright

The copyright statement. You should always fill this out and assign copyright to the correct author, group or corporation.

License

The software license(s) that this module can be licensed under. It’s enough to use an acronym like “GPL v2” or “MIT” or “Ms-PL” ... but if you have a custom license you can specify a description of it or the full text of it. Multiple entries are allowed.

LicenseUri

The url(s) for a full copy of the license. If you’re using an open source license you can link to it’s page on opensource.org

Category

We’ll publish our official category list later, but the category is intended to be an assignment to a single category in a hierarchical tree.

Tags

A collection of user-assigned free-text keywords that help to categorize the module and serve as additional search terms.

Homepage

If the module has a home page on the internet, this is the URL. You can use the link to the PoshCode Docs page (which our cmdlets will generate for you) if you don’t have your own page.

Download

This is the link to download the module. If you’re hosting it on PoshCode, we will fill this in. If you’re hosting it elsewhere, this must be a download link. If it’s a commercial module that requires payment, you should leave this link empty unless you have a free trial download. Note: I’m still working on the commercial angle. Would it be helpful to include the price here? If we offered a way to sell your modules through PoshCode, would you use it?

RequiredAssemblies

Assembly names (or relative paths, if the assemblies are included in the module folder) for assemblies that must be loaded before the module will work. These should be the Assembly FullName, not a partial name.

RequiredModules

The names and versions (and GUIDs) of any modules that this module depends on.

PowerShellVersion

The version of Powershell required (2.0, for now).

FrameworkVersion

The version of the .Net Framework required. The framework version implies the CLR version, so we’ll just stick to this. I’m still toying with the idea of saying the list is 2.0, 3.0, 3.5, 3.5.1, and 4.0 … rather than allowing the normal version with the full four digits.

VersionChanges

This is either the list of changes in this version, or a complete list of the changes in previous versions.

Exported Cmdlets

The list of (binary) cmdlets exported by this module.

Exported Functions

The list of script functions exported by this module.

Exported Aliases

The list of aliases exported by this module.

Exported Variables

The list of variables exported by this module.

Deprecated things …

I’m leaning against having these items in the PoshCode manifest — they don’t seem to be helpful for finding or sorting or understanding a module.

• ModuleFile
• ModuleToProcess
• FormatsToProcess

• TypesToProcess

Comments Requested

I’m hoping for feedback about this list, what I might be missing, or if there’s anything in here I should leave out … let me hear it!

As a warm up to writing my best-practices posts, I decided to answer this frequently asked question on the PowerShell wiki at PoshCode. I’m not going to repeat the whole post here, but suffice it to say that there’s a good explanation on the How Do I Install a PowerShell Module page, along with this script:

So, I’m building the next PoshCode around modules, with CPAN as my model …

Scripts are still there, but they’re still basically done the way they are now: you just upload a single file.

Modules, however, will be treated specially. You’ll have to have an account to upload, but you’ll be able to just upload a .zip file of your module folder using your login and not fill in any other forms. Our system will take care of parsing the metadata out of the manifest in your module folder.

The problem is that I need a little data that isn’t a part of the standard module manifest format… and I can’t add it because PowerShell won’t load a module with extra fields in the manifest metadata hashtable: https://connect.microsoft.com/PowerShell/feedback/ViewFeedback.aspx?FeedbackID=421837 — My request was closed “by design” early in the beta cycle and I wasn’t able to convince them to rethink that.

At a minimum, PoshCode requires a LICENSE field, and a CATEGORY field (think of the categories on CPAN or TechNet Script Center).

So I’m having an informal poll. (you’ll have to “vote” by commenting or tweeting to me, because I want more than just “I choose A”).

Which option do you prefer, or can you think of a better way:

Require the License and Category to be part of the PRIVATEDATA field in the standard manifest

This might require authors to rewrite parts of their modules, because we’d be forcing PrivateData to be a Hashtable, and to contain keys that they don’t need in the module.

Some existing modules use an array in PrivateData, or even a simple string, rather than a Hashtable. However, it’s not a huge difference, and might be the cleanest method: it wouldn’t require you to manage a second file of data, and the additional data will be easily available to users and scripts via the standard Get-Module output.

Add a “doc comment” system for Module manifest files.

The upside of this is that PoshCode wants to create module-level documentation anyway. If we use documentation comments like those used on functions we would be able to just create our own standard for them, and add any extra fields we need. It would be guaranteed not to conflict with anything you’re already doing, and in addition to missing metadata, you could have some module-level documentation beyond just the Description field of the metadata.

The problem with this is that there’s no built-in way to parse that, and doing so isn’t trivial, so you would need to just read it on our website, or read the source of the file, or have our PoshCode cmdlets in order to make any sense of it once you had a module on your system. It doesn’t integrate with PowerShell in any meaningful way.

Embed the extra data right in the manifest hashtable, using a special comment to hide it from Get-Module

@{
author=“Joel Bennett”
description=“My latest crazy module”
<#!PoshCode
License=“Ms-PL”
Category=“WPF”,“GUI”,“Toolkit” #>
CompanyName=“Huddled Masses”
...
}


I like this because it’s fairly trivial for us to strip out the comment and just turn that into a plain-old DATA section. It also lends itself to reminding the PowerShell team that these fields are missing, and maintains the existing simple syntax of the manifest.

The problem here is, again, that the data doesn’t appear using any of the standard PowerShell tools — but getting it out is significantly easier than getting out document comments…

Require a separate “ReadMe.psd1” (or “PoshCode.pds1” or “Metadata.psd1”) metadata file

You could start with a copy of your module manifest, and then add the extra stuff that PoshCode needs. This would be nice because we would be able to add any extra fields we wanted as mandatory members, and we could include the “documentation” stuff I mentioned earlier…

But the downside is that it would be a third file (second manifest) that authors would have to maintain and keep current.

Any thoughts? Suggestions?

I’ve put some further thoughts about CPAN and the data problem on the wiki.

Well, over the weekend I stole a few moments from thinking about PoshCode 2.0 to think about PoshCode 1.0 … and I added two things:

Feedback

I’ve replaced the misused “comment” box on the website with a full GetSatisfaction.com widget, so you can get your voice heard and get feedback (in the past it’s been impossible for me to respond to comments).

Twitter

PoshCode now posts new contributions to twitter on it’s own PoshCode account. Feel free to follow along. Of course, the information on there is severely truncated, by comparison with the PoshCode RSS feed … and only includes the link to the website, whereas the RSS feed also includes direct download links … but I figured some of you would appreciate it anyway, and I aim to please.

Of course, now in PoshCode 2 I’ll have to make sure we have users enter their twitter id’s in the profile page so we can be sure to cite you properly. Oh what a tangled web we weave…

I’ve updated the PoshCode script module to support CTP3, and added a -limit parameter to the Get-PoshCode cmdlet so you can specify how many items you want retrieved in the case where there are a lot of matches for your search terms — by default the limit is 25.

Improvements to the underlying web search API

You’ve always been able to pass a LIST parameter to the API, and get more results by specifying a higher number. But it never worked with the “path” notation (until now).

That is, you used to be able to do:

• http://poshcode.org/api1?q=start&list=10

• http://poshcode.org/api1?q=start&list=100

To make the API a little easier to use I’ve enhanced it just now:

1. You can now page the search results.
2. You can use the word “limit” instead of “list”
3. If you specify limit=0 (or list=0) I’ll give you everything I’ve got. Please use a little precaution about that, as it could be a LOT of data. I’d much rather you retrieve, say … 25, and then get the second page if you want more.

1. You can use path notation.

So, you can use any of these URLs:

• http://poshcode.org/api1/start/list/25/page/1
• http://poshcode.org/api1/start/limit/25/page/2
• http://poshcode.org/api1?q=start&list=25&page=3

• http://poshcode.org/api1?q=start&limit=25&page=4

There are a lot of search results for “start” ... feel free to play with enhancing the PoshCode module, or incorporating this into your apps, etc.

I should add that you don’t have to specify the limit or page number. By default you’ll get the first 10 items, which should be enough.

I keep forgetting to mention this! I’m speaking at the next PowerShell Virtual User Group this Thursday, November 13th at 12PM EST. Join us if you’re interested in any of my PowerShell CodePlex projects (PoshConsole, PoshCode, WASP). Actually, join us even if you’re not, because there’s going to be a demo of the latest version of PowerShell Plus by the folks from Idera, and Nathan Winters will be discussing managing Exchange with PowerShell.

Also, the PowerShell Community is running a survey about the possibility of a PowerShell convention, and what you’re interested in seeing at one… it’ll just take a moment

Well, an interesting comment came up today that the new version of PoshCode.org that we’re working on should support OpenSearch and of course, me being me, I said: oh, OpenSearch is easy, gimme a minute…

And three hours later, I’m going to bed.

However, I did manage to add paging support so you can “browse” the repository, and of course … to add OpenSearch. If you visit the repository with an OpenSearch aware browser, like say … Firefox 3, or IE 7, you’ll get a little glowing notification on your search bar that there is a search engine available. You can quickly add PoshCode’s search to your browser with a couple of clicks, and in no time flat you’re searching for PowerShell script.

Incidentally, I also hooked the OpenSearch stuff up to our API output, so OpenSearch aware apps which do not simply render the HTML page can get RSS output that should work for you. We’re not doing all of OpenSearch at this point, but I think the important bits are there (certainly the bits supported in your web browser are).

Feedback is welcome as always, let us know what you think of the new stuff by emailing feedback at poshcode …

So, there’s a few new features in the latest version of the PoshCode script module, and a few new features in the repository itself, including an embed feature, which I will demonstrate by embedding one here. Notice that the scripts on the site have an embed code, and it will normally embed the full height of the script, but you can add a height parameter as I did here to force the height and cause vertical scrolling.

Resolve URL

New Cmdlet features

The latest version of both the v1 compatible PoshCode script (you have to dot-source it) and the v2 CTP2 compatible module support an -Upgrade parameter to the Get-PoshCode script, so you can call Get-PoshCode -Upgrade and it will retrieve the latest version of the script (whether you need it or not).

The New-PoshCode script has been fixed so you can use it to submit to other pastebin sites (particularly, the temporary one we use for support on IRC).

New Site features

The searches now return items with the same “probability” of being correct in date order, so newer versions of scripts show up at the top — this also affects the cmdlets, and the integration with apps such as PowerGUI. Note, however, that if an older version of the script fits your search-terms better, it may still show up at the top — I’m working on this.

The PoshCode site now supports wrapping by other community sites — if you want to wrap our site in yours, let me know, and I can give you a custom subdomain and allow you some control over the display, as I did for PowerShell Community …

I finally fixed the problem with logging that my host was having, so I have a week’s worth or so of logs which Webalyzer informs me show that we average about 600 visits a day, and a surprising number of subscribers to our RSS feed…

Upcoming features

I have three new features planned for the next couple of weeks or so: finishing the browsing feature, implementing better version-tracking, and adding tagging. The key feature missing from the brownsing is the ability to go to the “next page” of results (right now you can only see the first page of scripts for the site or for authors) — we should have that fixed soon.

The version tracking feature (which is mostly a matter of cleaning up some db queries) will result in being able to build a visual history for a given submission, including branching, etc. and should allow users to easily retrieve “the latest” version of a given script.

Tagging will improve searching and browsing by providing a way to categorize scripts and browse by tags, but I’m a little nervous about it because it seems that to be useful it will need to allow tags to be added by users who didn’t create the original script, and this is complicated since we have not so far created a login/authentication system.

I’m still debating how to properly do this, but I think that ultimately we’re going to have to use a login system with support for OpenID so that authors can identify themselves with their blog URLs, etc … and can use the same identity on the PowerShell Community site once that site gets their OpenID implementation working properly.

The alternative that I had been considering: using signed scripts as the only authentication, seems to fail in the instance of tagging — it would basically require you to make up a PSD1 file specifying the script id and the tags you want to apply and then sign it and submit it. I think you’ll agree that’s way too much work for the end users, even if it was automated — and it would also require the server to validate the signature and parse the script!

I’ve been having problems with the search functionality on the PoshCOde repository, and I just thought I’d throw this up here because I just now solved the biggest problem: ranking. Up until now, the results have not been returned in order of relevance — this is because the search works using MySQL’s FULLTEXT BOOLEAN search, which doesn’t return in relevance order, nor does it return an extra ‘score’ column.

I’ve fixed that, and weighted the search so that words in the title count more than words in the code by creating a relevance column by hand:

Incidentally, the FULLTEXT index means that words shorter than 4 characters don’t count (I’m going to try to get this changed, but it’s an option for MySQL, so it has to be changed in the config file) in the meantime you can search for words using the wildcard character, like: SQL* and it sort-of works. The PoshCode cmdlet actually was adding *‘s to the query (although I’ve just decided that’s not a good idea, because it means that queries from the cmdlet appear to have different results than queries on the website.

MySQL’s FULLTEXT BOOLEAN search has all sorts of features (and limitations): there is a stopword list, maximum and minimum word lengths, and all sorts of operators for setting word precedence or negating words, or weighting them negatively … to REQUIRE that a word be present, it must have a + in front, and in order to mark a word as more important, you have to put > in front, not just put it first… I’ve been thinking about trying to apply a few of those tricks myself (eg: put * on words under four characters, and put > on the first 30% of words and < on the last 30% to try to simulate weighting them …) but my original feeling was that the search is more powerful if you just know that it’s a fulltext boolean search and can write your queries accordingly.

If anyone has any ideas for how to improve search in MySQL … or opinions on whether I should try to apply boolean operators to queries which don’t already have them … please let me know.